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EXECUTIVE  SUMMARY 


For  three  consecutive  years,  the  Federal  Aviation  Ad¬ 
ministrations  (FAAs)  Office  of  Aviation  Safety  (AVS) 
Chief  Scientific  and  Technical  Advisory  (CSTA)  program, 
and  the  Human  Factors  Division  of  the  Civil  Aerospace 
Medical  Institute  (CAMI)  have  conducted  an  annual 
workshop  dedicated  specifically  to  maintenance  and 
engineering.  The  2012  workshop  reported  here  addressed 
both  problems  and  solutions  associated  with  technical 
documentation  for  maintenance. 

Twenty-eight  invited  attendees  came  from  government, 
research  and  development,  manufacturing,  airlines,  and 
maintenance,  repair,  and  overhaul  organizations.  The  first 
CSTA  workshop  on  maintenance  human  factors  issues,  in 
2010,  identified  technical  documentation  as  the  number 
one  human  factors  challenge  in  aviation  maintenance 
(Johnson,  20 10;  Avers,  Johnson,  Banks,  &Nei,  201 1).  At 
the  2012  workshop,  one  attendee  noted  “ ...  it  is  known 
that  the  technical  documentation  challenge  is  the  great¬ 
est  risk  in  the  aviation  industry  -  it  will  take  more  than 
a  scientist’s  workshop  to  fix  the  issues,  but  this  is  a  good 
start.”  Issues  associated  with  technical  documentation  are 
known  to  cause  errors,  rework,  maintenance  delays,  other 
safety  hazards,  and  FAA  administrative  actions  against 
individuals  and  organizations.  In  National  Aeronautics 
and  Space  Administration  (NASA)  studies  of  Aviation 
Safety  Reporting  System  (ASRS)  reports,  45  to  60%  of 
incidents  were  procedure  related  or  involved  technical 
documentation  (Kanki  &  Walter,  1997).  These  instances 
threaten  safety  and  cost  the  industry  millions  of  dollars. 

The  workshop  format  combined  key  presentation 
topics,  followed  by  structured  discussion  and  small  group 
exercises.  It  began  by  clarifying  issues  regarding  technical 
documentation  for  maintenance  and  ended  with  exten¬ 
sive  lists  of  challenges  and  corresponding  short-term 
and  long-term  solutions,  rank-ordered  by  priority.  The 
report  describes  the  group  processes  and  data  collection 
technique  used  to  identify  the  top  ten  industry  action 
items  for  addressing  documentation  issues: 

1.  Quantify  financial  loss  related  to  documentation 
issues. 

2.  Develop/apply  methods  for  evaluating  quality  of 
technical  documentation. 


3.  Leverage  voluntary  reporting  to  identify  specific 
problems  with  documentation. 

4.  Improve/create  guidance  for  FAA  personnel  work¬ 
ing  documentation  issues,  especially  Instructions  for 
Continued  Airworthiness  (ICA). 

5.  Expand  incident  investigation  to  identify  details 
associated  with  documentation  issues. 

6.  Improve  integration  and  linkage  of  content  across 
maintenance  documents  —  maintenance  manuals, 
task  cards,  and  illustrated  parts  catalogs. 

7.  Delegate  approval  from  FAA  to  industry  using  es¬ 
tablished  Organization  Designation  Authorization 
(ODA). 

8.  Improve  usability  of  manual  format,  accessibility  of 
manual,  and  training  on  manual  use. 

9.  Initiate  industry  mandate  requiring  users  to  address 
known  documentation  issues. 

10.  Improve  coordination  of  document  professionals 
from  industry  segments  and  government. 

The  workshop  design  reflected  the  importance  of 
government  and  industry  working  together  in  a  concerted 
effort  to  prioritize  solutions  for  improving  maintenance 
documentation.  Attendees  acknowledged,  on  day  1  of 
the  workshop,  that  more  of  the  same  “talking  about  the 
challenges”  is  insufficient  action.  All  segments  of  indus¬ 
try  and  government  must  make  an  organizational  and 
financial  investment  to  address  documentation  problems. 

The  technical  documentation  challenge  is  complex  due 
to  organizational  and  regulatory  processes,  technological 
innovations,  and  design  quality,  to  name  a  few.  This  makes 
solution  implementation  difficult.  Fixing  the  problems 
means  changing  a  culture.  The  question  remains:  Is  the 
aviation  industry  ready  to  tackle  these  challenges? 

The  development  of  viable  solutions  is  a  shared  respon¬ 
sibility  that  requires  open  communication  from  all  of  the 
stakeholders  —  aviation  maintenance  technicians  (AMTs) , 
AMT  supervisors,  corporate  executives,  manufacturers, 
suppliers,  and  government.  Without  a  collective  effort, 
technical  documentation  issues  will  continue  to  be  a 
safety  risk.  This  workshop  and  report  are  important  first 
steps  to  taking  action. 
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Technical  Documentation  Challenges  in  Aviation  Maintenance: 

A  Proceedings  Report 


section  i.o  workshop 
proceedings 

1.1  Background  on  Technical  Documentation  Issues 

“The  technicians  failed  to  follow  the  written  procedures  ...” 

This  statement  is  often  found  in  descriptions  of  minor 
maintenance  errors  in  National  Transportation  Safety 
Board  (NTSB)  reports  of  major  aircraft  accidents.  Written 
procedures  refer  to  a  variety  of  manufacturer  publications, 
specific  company  job  cards,  or  the  rule  in  Code  of  Federal 
Regulations  (CFR)  14  Part  43  Section  43.13(a)  entitled, 
“Acceptable  Methods,  Techniques,  and  Practices —Aircraft 
Inspection  and  Repair.”  The  rule  states  that: 

. .  .each  person  performing  maintenance,  alteration,  or 
preventative  maintenance  on  an  aircraft,  engine,  propeller, 
or  appliance  shall  use  the  methods,  techniques,  and  prac¬ 
tices  prescribed  in  the  current  manufacturer’s  maintenance 
manual  or  Instructions  for  Continued  Airworthiness  pre¬ 
pared  by  its  manufacturer,  or  other  methods,  techniques, 
or  practices  acceptable  to  the  Administrator. 

The  rule  is  clear:  Use  a  manual  for  all  work.  It  should 
be  easy,  but  it  is  not  straightforward.  Figure  1  represents 
the  volume  of  manuals  or  data  delivered  to  operators  of 


multiple  aircraft  types  in  support  of  in-service  operations 
and  maintenance.  Even  carrying  the  documentation  for 
a  single  task  may  become  unwieldy  when  maintenance  is 
performed  in  a  restricted  space  or  at  night  on  the  ramp. 
While  the  pile  of  paper  could  be  very  high,  most  airlines 
receive  and  use  the  majority  of  their  documentation  in 
digital  format. 

The  volumes  of  documentation  make  it  easy  to  un¬ 
derstand  how  documentation  problems  compound  in 
Maintenance,  Repair,  and  Overhaul  organizations  (MROs) 
where  each  aircraft  must  be  maintained  by  maintenance 
documentation  specific  to  the  aircraft  owner  and  registry 
number.  This  means  that  an  MRO  is  likely  to  have  different 
work  instructions  for  each  aircraft  in  the  shop,  even  though 
they  are  the  same  type  of  aircraft.  Most  air  carrier  opera¬ 
tions  supplement  manufacturers  manuals  with  company 
instructions,  checklists,  job  cards,  and  more;  and  some 
companies  use  computer-based  maintenance  documents 
with  varying  degrees  of  user  friendliness. 

The  maintenance  documentation  issues  extend  to  the 
smaller  general  aviation  (GA)  aircraft  and  to  all  aircraft 
components  but  can  manifest  in  different  ways.  GA 
operators  often  lack  some  of  the  standardization  and 
documentation  that  is  required  in  air  carrier  operations. 


Copyright  2012  The  Boeing  Company.  All  rights  reserved.  Reproduced  courtesy  of  The  Boeing  Company. 

Figure  1.  The  technical  documentation  for  Boeing  aircraft  model. 
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GA  technicians  report  there  is  not  enough  technical  docu¬ 
mentation  for  some  aircraft.  Regardless,  all  technicians 
are  responsible  for  integrating  maintenance  instructions 
from  multiple  sources  (e.g.,  manufacturer  manuals,  service 
bulletins,  and  airworthiness  directives)  -  a  situation  that 
can  make  documentation  a  significant  challenge  for  those 
who  maintain  aircraft. 

Before  presenting  details  of  the  workshop,  here  are 
examples  (Giustozzi,  2009)  of  the  scope  of  the  technical 
documentation  issue: 

In  2000,  an  FAA  study  looked  at  maintenance  error. 
The  study  focused  on  major  malfunctions  that  occurred 
within  90  days  of  a  heavy  maintenance  check.  Failure  to 
comply  with  maintenance  documentation  was  the  number 
one  reason  for  malfunction  (Johnson  &  Watson,  2001). 

In  2004,  the  NTSB  accident  report  of  the  Charlotte 
USAir  Express  Accident  (AAR-04-01)  stated  that  the 
FAA  should:  “...require  14  CFR  Part  121  air  carriers 
to  implement  a  program  in  which  carriers  and  aircraft 
manufacturers  review  all  work  card  and  maintenance 
manual  instructions  for  critical  flight  safety  systems  and 
ensure  the  accuracy  and  usability  of  these  instructions  so 
that  they  are  appropriate  to  the  level  of  training  of  the 
mechanics  performing  the  work. 

In  2007,  a  report  by  the  Confidential  Human  Factors 
Incident  Reporting  Programme  (CHIRP)  from  the  United 
Kingdom  (www.chirp-mems.co.uk)  indicated  the  top 
two  most  frequently  occurring  errors  reported  were:  (1) 
information  not  used  and  (2)  procedures  not  followed. 


Their  recommendation  was  to  simplify  the  procedures  and 
align  company  task  cards  with  the  aircraft  maintenance 
manual  (Rankin,  2008). 

A  2012  analysis  of  the  FAA  enforcement  database  for 
actions  taken  against  mechanics  regarding  14  CFR  Part 
43  Section  43. 1 3(a)  showed  that  technical  documentation 
is  a  challenge.  Of  nearly  900  “closed”  cases  from  2010, 
more  than  850  actions  were  taken  against  mechanics. 
Over  one- third  of  the  violations  (36%)  were  associated 
with  not  using  the  proper  technical  documentation.  The 
data  revealed  this  is  the  number  one  cause  for  Enforce¬ 
ment  Investigation  Reports  (S.  Hodges-Austin,  personal 
communication,  April  4,  2012). 

A  2012  analysis  of  the  NASA  ASRS  maintenance 
reports  from  2001  to  201 1  (14,267  reports)  showed  that 
nearly  64%  (about  9,000)  of  safety  incidents  coded  in  the 
reporting  system  were  related  to  technical  documenta¬ 
tion  or  procedural  challenges  or  both  (J.  Moya,  personal 
communication,  March  29,  2012). 

1.2  Workshop  Attendees 

The  workshop  planners  invited  participants  who  had  a 
stake  in  the  issue  of  technical  documentation  for  aviation 
maintenance.  All  28  attendees  possessed  considerable 
expertise  from  either  operations  or  science,  including 
MROs,  Original  Equipment  Manufacturers  (OEMs), 
airlines,  FAA  offices,  research  and  development  (R&D), 
and  third-party  data  management  providers  (See  Figure  2) . 

1.3  Workshop  Format 

The  workshop  format  fostered  participant  interactions, 
application  of  analytical  methods,  and  a  multi-disciplinary 
approach  to  addressing  problems  associated  with  techni¬ 
cal  documentation  in  the  aviation  maintenance  industry. 
The  format  employed  individual,  small-group,  and  large- 
group  participative  techniques.  There  were  12  formal 
presentations  divided  into  session  topics,  following  the 
keynote  speaker  and  individual  attendee  introductions. 
(See  presentation  slides  in  Appendix  B.)  Select  attendees 
led  a  solution-oriented  group  discussion  at  the  end  of 
each  session.  Following  all  presentations,  five  working 
groups  were  formed  to  identify  technical  documentation 
challenges  and  corresponding  short-term  and  long-term 
solutions  within  one  of  the  designated  focus  areas:  (1) 
document  quality,  (2)  measurement,  (3)  user/mechanic, 
(4)  government,  and  (5)  industry/management.  Each 
working  group  presented  their  lists  of  challenges  and 
solutions  to  the  full  workshop  group.  At  the  end  of 
the  workshop,  attendees  provided  an  evaluation  of  the 
workshop. 


Figure  2.  A  depiction  of  attendee  affiliation  (count 
and  percentage). 
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1.4  WORKSHOP  -  DAY  1 

The  two-day  workshop  was  held  in  Atlanta  at  the  Flight 
Standards  District  Office  (see  Figure  3).  The  following 
subsections  summarize  the  workshop  presentations  and 
activities. 


Figure  3.  Dr.  Bill  Johnson  thanks  Mr  Keith  Frable 
for  hosting  the  workshop  location. 

1.4.1  Welcome  Session 

Dr.  William  (Bill)  Johnson,  CSTA  for  Maintenance 
Human  Factors,  opened  the  meeting  and  welcomed  the 
attendees.  He  summarized  the  accomplishments  of  the 
last  maintenance  human  factors  workshop  and  identi¬ 
fied  four  objectives  for  the  current  meeting.  He  asked 
workshop  attendees  to: 

•  Identify,  summarize,  and  prioritize  the  technical  docu¬ 
mentation  problems  in  operational  terms, 

•  Estimate  the  affect  of  technical  documentation  issues 
on  aviation  safety  and  efficiency, 

•  Identify  and  differentiate  short-term  and  long-term 
solutions  for  technical  documentation  problems,  and 

•  Create  actionable  guidance  for  the  FAA,  research  and 
development  community,  and  industry. 

1 .4.2  Keynote  Address  “Why  are  we  talking  about 
technical  publications?” 

Caroline  Daniels  —  Aircraft  Technical  Publishers 
Ms.  Caroline  Daniels,  Chairwoman  of  the  Board  and 
Chief  Executive  Officer  of  Aircraft  Technical  Publishers 
(ATP),  is  widely  recognized  as  a  pioneer  of  safety  infor¬ 
mation  management  systems.  Her  company  is  involved 
in  issues  surrounding  technical  documentation  in  avia¬ 
tion  maintenance.  In  her  opening  statement,  she  asked 
attendees  to  consider  technical  documentation  a  shared 
responsibility.  As  an  industry,  we  all  have  the  responsi¬ 
bility  to  document  and  inform,  mitigate  risk,  establish 
processes,  and  meet  regulatory  requirements  pertaining  to 


technical  documentation.  To  do  this  effectively,  we  must 
consider  the  complete  lifecycle  of  technical  publications. 
She  went  on  to  say  that  each  maintenance  organization 
must  make  its  own  unique  set  of  technical  documenta¬ 
tion  requirements,  considerations,  and  decisions.  At  each 
point  in  the  development  process,  decisions  have  to  be 
made  regarding  authoring  (tools),  printing  (media),  and 
distribution  (type  of  reader).  These  decisions  are  critical 
and  can  produce  very  different  results  for  the  end  user, 
depending  on  the  authoring  tools  and  formatting  selected. 

Cost  influences  document  development.  Costs  de¬ 
termine  how  a  company  creates  the  complete  technical 
publication  lifecycle  to  support  its  company  strategy.  There 
are  low  cost  resources  available  (e.g.,  Microsoft®  Word 
for  authoring  and  Adobe®  pdf  for  delivery),  but  they  are 
not  cost-effective  in  the  long  term  because  the  content  is 
unstructured  and  difficult  to  search  and  maintain.  Based 
on  the  experience  of  ATP,  structured  authoring  and  well- 
crafted  document  architecture  produce  the  best  results 
for  the  money.  This  is  not  easily  accomplished;  it  requires 
oversight  and  administration,  strict  discipline  among  the 
authors,  and  extensive  information  technology  structure 
and  support. 

Currently,  there  is  no  regulatory  specification  regarding 
media,  format,  turn-around  times,  or  distribution  tech¬ 
nology  for  technical  documentation.  We  are  at  a  point 
where  we  can  utilize  next-generation  technologies  and 
provide  new  opportunities  for  presenting  and  accessing 
technical  information  (e.g.,  3D  modeling,  embedded 
video  training,  and  voice  recognition).  Even  with  these 
new  technological  capabilities,  we,  as  an  industry,  have 
to  focus  on  “delivering  the  right  form  and  right  pieces 
of  information  into  the  right  hands,  at  the  right  time 
and  place.” 

1 .4.3  Workshop  Introductions 

Next,  attendees  introduced  themselves  to  the  group 
and  presented  what  they  considered  as  the  most  important 
problem  relating  to  technical  documentation  in  aviation 
maintenance  and  offered  three  viable  solutions  to  over¬ 
come  the  problem.  Some  attendees  provided  multiple 
problems  and  corresponding  solutions,  while  others 
provided  one  of  each.  We  had  requested  each  attendee 
submit  problem  and  solutions,  prior  to  the  workshop. 

The  content  analysis  conducted  on  the  problems 
and  solutions  included  input  from  28  attendees.  They 
submitted  79  problems  and  80  solutions,  which  were 
independently  analyzed  based  on  coding  protocols  de¬ 
veloped  for  each  (Sutton,  2010;  Walker,  2011).  Three 
raters  separately  applied  the  protocols  and  then  met  to 
address  coding  discrepancies  before  coming  to  an  agree¬ 
ment.  Their  results  were  presented  to  the  group  on  the 
second  day  of  the  workshop. 
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Figure  4.  Distribution  of  identified  problems  within 
categories  (count  and  percentage). 
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Figure  5.  Distribution  of  proposed  solutions  within 
categories  (count  and  percentage). 


1.4.3. 1  Proposed  Problems 

Results  of  the  problem  content  analysis  (Figure  4), 
reveal  that  prior  to  the  workshop  nearly  half  (48%)  of 
the  identified  problems  with  technical  documentation 
are  attributable  to  the  processes  involved  in  producing 
and  sustaining  quality  end-products.  Twenty-eight  per¬ 
cent  of  the  problems  reveal  issues  AMTs,  as  end  users, 
face  with  low-quality  documentation.  The  remaining 
24%  of  problems  reveal  features  of  an  organization  that 
contribute  to  documentation  issues.  Figure  4  emphasizes 
the  broad  range  of  problems  associated  with  technical 
documentation  improvement. 

1. 4.3.2  Proposed  Solutions 

Results  of  the  solutions  content  analysis  (Figure  5), 
reveal  that  prior  to  the  workshop,  36%  of  the  proposed 
solutions  would  involve  changes  in  processes  (incorpo¬ 
ration  of  analytics  and  leveraging  technology),  37%  in 
people  (qualifications,  training,  and  cultural  shift),  and 
27%  in  products  (standards  and  guidelines,  and  techni¬ 
cal  manuals).  Again,  this  shows  many  solutions  were 
proposed,  but  a  silver  bullet  is  unlikely. 

1.4.4  Day  1  Presentations 

The  two  presentation  sessions  (Figure  6)  conducted 
the  first  day  of  the  workshop  were  entitled:  “Summariz¬ 
ing  Government  and  Industry  Challenges”  and  “Creating 
and  Delivering  Technical  Documents.” 


Figure  6.  Workshop  presentations  -  Day  1. 
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1.4.4. 1  Session  1:  Summarizing  Government  and 
Industry  Challenges 

Keith  Frable  —  Federal  Aviation  Administration 

Mr.  Keith  Frable,  Principal  Maintenance  Inspector 
for  Delta  Air  Lines,  Certificate  Management  Office,  was 
involved  with  the  merger  between  Delta  and  Northwest 
Airlines  and  has  been  employed  by  the  FAA  for  more  than 
1 5  years.  Mr.  Frable  offered  a  detailed  overview  of  the  Air 
Transportation  Oversight  System  (ATOS).  Under  ATOS, 
the  FAAs  primary  responsibilities  are  to  1)  continually 
validate  the  performance  of  an  air  carrier’s  approved  and 
accepted  programs  for  the  purpose  of  continued  opera¬ 
tional  safety,  and  2)  conduct  performance  assessments  to 
confirm  that  the  air  carrier’s  operating  systems  produce 
intended  results.  Aviation  Safety  Inspectors  (AS Is)  use 
other  data  to  assess  the  air  carrier’s  operational  health, 
including  the  Voluntary  Disclosure  Reporting  Program 
(VDRP),  the  Aviation  Safety  Action  Program  (ASAP),  and 
the  Safety  Management  System  (SMS).  These  program 
databases  provide  the  opportunity  to  identify  and  address 
operational  safety  issues  without  punitive  damages.  In 
the  operations  overseen  by  Mr.  Frable,  the  majority  of 
disclosures  reported  are  categorized  as  a  failure  to  follow 
or  adhere  to  approved  documentation  or  procedures. 
He  noted  the  documentation  was  seldom  incorrect 
and  resulted  in  a  failure.  When  there  are  slight  errors, 
the  mechanic  often  recognizes  the  issue  and  triggers  an 
engineering  change  to  the  document. 

The  challenge  for  many  AS  Is  is  the  magnitude  of 
data.  Inspectors  receive  data  from  many  sources,  and  it 
is  difficult  to  absorb  all  of  the  data  points  and  develop 
a  “silver  bullet”  approach.  The  principal  maintenance 
inspectors  are  consistently  trying  to  answer  the  question, 
“How  do  we  fix  this  problem?”  and  “What  regulatory 
solutions  can  be  provided?” 

Bill  Norman  —  TIM  CO  Aviation  Services 

Mr.  Bill  Norman,  President  of  MRO  services  for 
TIMCO  Aviation  Services,  is  responsible  for  TIMCO’s 
Airframe,  Engine  and  Line  Care  Maintenance  businesses. 
Mr.  Norman  offered  insight  into  the  issues  repair  stations 
and  air  carriers  struggle  with  regarding  technical  data 
for  aviation  maintenance.  Technical  data  are  rarely  all  in 
one  place.  Technicians  must  assimilate  information  from 
multiple  parts  of  a  manual  and  even  multiple  manuals. 
Moreover,  they  are  required  to  pull  all  reference  data  for  a 
given  task  and  wade  through  the  data  to  determine  exactly 
what  is  and  is  not  applicable  to  the  maintenance  task. 

Mr.  Norman  shared  an  example  of  an  accident  where 
an  A-320  departed  the  runway  due  to  a  cross-wired  anti¬ 
skid  transducer  connector.  In  this  accident,  technical 
documentation  played  a  significant  role.  The  technical 


data  required  to  perform  work  on  the  anti-skid  transducer 
connector  included  one  airworthiness  directive  reference, 
1 1  service  bulletin  (SB)  references,  and  62  customer  task 
card  AMM  (Aviation  Maintenance  Manual)  references. 
This  resulted  in  73  references  total  and  6.25  pounds  of 
reference  data  the  technician  had  to  go  through  to  com¬ 
plete  the  task.  In  addition  to  the  quantity  of  technical 
data  tied  to  this  task,  the  Airbus  diagram  did  not  follow 
aviation  norms.  The  air  carrier  developed  the  job  card 
using  the  “do  per  AMM  XX-YY-ZZZ”  approach.  “Do 
per”  is  a  shortened  instruction  to  follow  a  reference  listed 
elsewhere  in  the  documentation  or  associated  company 
job  card.  Instead  of  usable  stepped  work  instructions, 
the  job  card  sequencing  was  reverse-ordered  in  places, 
the  transducer  electrical  connectors  were  not  uniquely 
keyed,  and  the  final  test  of  the  anti-skid  system  was  inad¬ 
equate.  All  of  these  issues  (minus  aircraft  design  for  the 
connectors)  could  be  addressed  with  improved  guidance 
from  the  manufactures  and  improved  Instructions  for 
Continued  Airworthiness  (ICAs). 

Clear  and  easy-to-understand  ICAs  are  needed  now 
and  will  continue  to  be  needed  as  the  industry  experi¬ 
ences  significant  changes  in  the  type  of  technicians  (non- 
certificated)  available  for  hire  and  the  type  of  aircraft 
being  repaired  and  maintained. 

John  Hall  —  International  Association  of  Machinists 

Mr.  John  Hall,  lead  mechanic  and  Director  of  the 
International  Association  of  Machinists  and  Aerospace 
Workers  (IAMAW)  Flight  Safety  Committee,  serves  on 
numerous  industry  and  regulatory  committees  involving 
aviation  maintenance  safety.  Mr.  Hall  reviews  not  only 
the  technical  documentation  problems  as  they  relate 
specifically  to  the  mechanic  user  but  also  the  process  that 
is  followed  for  corrective  action.  Once  a  mechanic  com¬ 
mits  an  identifiable  error,  that  mechanic  receives  a  letter 
of  investigation  from  the  FAA.  The  mechanic  notifies 
the  company’s  field  safety  representative  and  meets  to 
establish  the  facts.  The  flight  safety  representative  then 
calls  the  principal  maintenance  inspector  and  requests 
an  informal  meeting  to  discuss  the  issue  with  the  ASI. 
When  a  violation  is  identified,  the  field  safety  representa¬ 
tive  presents  any  factors  involved  in  the  commission  of 
the  error  and  seeks  remediation  for  the  mechanic.  Some 
of  the  contributing  factors  identified  in  this  process 
include  confusing  or  conflicting  paperwork  and  issues 
with  equipment  and  shop  procedures. 

Mr.  Hall  noted  that  technical  documentation  continues 
to  be  a  problem  for  mechanics.  At  U.S.  Airways  in  the 
last  10  months,  approximately  35%  of  the  ASAP  reports 
were  paperwork  related  (e.g.,  failure  to  follow,  confus¬ 
ing,  and  conflicting)  and  10%  of  the  reports  resulted  in 
changes  to  paperwork.  He  identified  some  possible  solu- 
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tions  as  including  recurrent  training,  crew  briefings,  alert 
bulletins,  employee  involvement  in  changes  to  technical 
publications,  and  user-friendly  electronic  publications. 
In  ASAP  InfoShare  meetings,  many  carriers  suggest  that 
paperwork  issues  contributes  to  as  much  as  70%  of  the 
reports. 

Brad  Shelton  —  Delta  Air  Lines 
Mr.  Brad  Shelton,  the  Managing  Director  of  Engi¬ 
neering,  Quality,  Technology  and  Training  for  Delta  Air 
Lines,  is  responsible  for  fleet  and  propulsion  engineering, 
maintenance  programming,  reliability,  publications,  and 
technical  records.  Mr.  Shelton  opened  his  presentation 
by  stating,  “We  are  ultimately  responsible  for  the  work 
accomplished  on  our  aircraft.”  This  responsibility  is 
directly  influenced  by  the  document  usability  and  tech¬ 
nical  accuracy.  To  appropriately  mitigate  and  eliminate 
technical  documentation  issues,  we  all  (industry,  OEMs, 
and  regulators)  must  work  together. 

Delta  pursues  four  key  objectives  relating  to  documen¬ 
tation:  1)  provide  accurate  and  consistent  data,  2)  reduce 
opportunities  for  human  error,  3)  maintain  regulatory 
compliance,  and  4)  effectively  and  efficiently  manage 
data.  In  pursuing  these  objectives,  Delta  faces  significant 
challenges  in  terms  of  data  consistency,  data  volume, 
data  complexity,  data  re-use,  and  data  delivery  (Figure 
7).  Delta  is  pursuing  multiple  solutions  to  address  each 
problem.  To  improve  document  usability  in  the  future, 
the  industry  must  take  action  in  five  areas: 

•  Manufacturers  must  invest  in  quality  and  data  consis¬ 
tency  among,  and  within,  their  products. 

•  Manufacturers,  operators,  and  delivery  system  develop¬ 
ers  must  invest  in  the  linked  tools  that  cross  all  content. 

•  Content  delivery  providers  must  have  “app  like”  sim¬ 
plicity  and  intuitive  user  interfaces. 
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-  Delta  manages  about  3  million  pages  of  data 

*  About  10%  of  the  content  is  revised  annually 

-  About  10%  of  the  content  is  customized  by  the  airline 

*  About  5%  of  the  content  is  re-used  for  work  cards  and  other 
ma  i  nte  na  nee  d  oc  u  me  n  ts 

Our  Role  and  Response 

-  Processes  for  document  quality  include  check  and  approve 

*  Regularly  monitor  throughput,  backlog  and  quality  metrics 

-  Leverage  automation  for  processing  OEM  revisions 

*  Drive  custom  content  back  to  the  OEM  for  inclusion  as  basic 


Mte  T«hQpS  I  F«p6o»rv  2012  |  7 


Figure  7.  Delta’s  data  volume  statistics  and  corpo¬ 
rate  response. 

•  Airlines  and  operators  must  instill  confidence  in  their 
end  users  that  the  data  are  correct 

•  Regulators  must  have  controls  for  consistency  and 
incorporation  of  required  content  by  OEM. 

Ultimately,  we  must  recognize  that  there  are  a  num¬ 
ber  of  obstacles  to  providing  useable  documentation  for 
aircraft  maintenance.  Even  though  we  have  interven¬ 
tions  in  place,  we  must  continue  to  look  toward  future 
methods  of  improvement.  The  industry  and  regulatory 
agencies  must  collaborate  to  reduce  and  remove  current 
and  future  challenges. 

Brockford  Tubbs  —  The  Boeing  Company 
Mr.  Brockford  Tubbs,  a  manager  in  the  Maintenance 
Engineering  Group  for  the  Boeing  Company,  is  respon¬ 
sible  for  leading  the  Airplane  Maintenance  Manual  Us¬ 
ability  Team,  managing  the  737  model,  and  supervising 
maintenance  engineering  personnel  for  all 
heritage  Boeing  models.  Mr.  Tubbs  focused 
on  the  capabilities  of  e-enabled  business 
solutions.  The  challenge  today  in  the  infor¬ 
mation  age  is  to  work  through  processes  to 
implement,  activate,  and  integrate  systems 
for  future  technologies  while  using  the 
source  information  as  the  foundation  for 
making  improvements. 

In  the  past,  delivering  a  maintenance 
manual  revision  was  no  simple  feat  (Figure 
8).  Before  PMA  (Portable  Maintenance 
Aids),  MyBoeingFleet.com  (fast,  real-time 
delivery)  and  Maintenance  Performance 
Toolbox  (improved  linkage  and  user  ex¬ 
perience),  revising  and  delivering  technical 
support  information  was  a  time-consuming, 
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1,130  Operators 

•  Maintain  2,532  Active  Manuals  for: 

707/720, 727,  737,  747,  757, 767,  777 

•  Maintain  43,071 ,266  Different  Pages  in  These  Manuals 

•  Revise  Most  of  These  Manuals  Every  120  Days 


•  Maintain  5,500  Active  Manuals  for: 

707/720, 727,  737,  747,  757,  767, 777, 787,  P8A 

•  Maintain  184,068,439  Different  Pages  in  These  Manuals 

•  Revise  Most  of  These  Manuals  Every  120  Days 
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Figure  8.  Boeing’s  technical  documentation  transition  over  10  years. 
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Figure  9.  Boeing's  transition  from  paper  technical  data  to  PDA-accessible  data. 


expensive,  and  slow  process.  The  resources,  time,  and 
energy  involved  required  millions  of  dollars,  mountains  of 
paper,  and  untold  amounts  of  microfiche.  Today,  Boeing 
is  able  to  make  and  deliver  changes  almost  in  real  time 
using  improved  authoring  and  web-based  delivery  tools 
(see  Figure  9).  In  addition,  Boeing  continues  to  develop 
new  tools  and  services  to  enhance  the  usability  of  their 
products.  Maintenance  Performance  Toolbox  service  of¬ 
ferings,  which  include  enhanced  graphics,  linking,  and 
navigation,  are  examples  of  how  industry  is  changing  the 
way  end  users  interact  with  OEM  technical  data. 

As  Boeing  looks  toward  the  future,  visual  graphic 
content  is  on  the  horizon.  Boeing  is  considering  how  to 
enhance  online  tool  usage.  One  improvement  is  real-time, 
easily  accessible  maintenance  information.  For  example, 
in  the  new  system  a  mechanic  would  get  intuitive  visual 
graphic  information  about  where  to  look,  what  to  re¬ 
move,  and  how  to  process  a  procedure.  Boeing  plans  to 
link  visual  tools  with  the  actual  written  manual  content. 

Ultimately,  Boeings  goal  is  to  get  the  right  information 
to  the  right  people,  at  the  right  time,  all  while  making 
decisions  optimized  for  operational  safety,  security,  and 
compliance  standards.  Boeing  continues  to  keep  the 
customer’s  experience  in  mind  and  keep  ties  to  their  user 
community  by  providing  information  in  a  usable  format 
where  safety  and  repeatability  are  paramount. 

Discussants 

The  Honorable  John  Goglia  and  Dr.  Bill  Rankin  led 
the  group  discussion  at  the  end  of  the  session  on  summa¬ 
rizing  government  and  industry  challenges  and  solutions. 


1. 4.4.2  Session  2:  Creating  and  Delivering 
Technical  Documents 

Bill  Colleran  —  Gulfstream  Aerospace  Corporation 

Mr.  Bill  Colleran,  Director  of  Technical  Information 
Services  for  Gulfstream  Aerospace  Corporation,  is  respon¬ 
sible  for  developing  and  sustaining  aircraft  publications 
for  all  Gulfstream  model  aircraft,  fleet  computerized 
maintenance  programs,  and  aircraft  maintainability  and 
reliability  programs.  Mr.  Colleran  noted  the  history  of 
technical  documentation  and  that  its  transition  is  evolving 
very  rapidly.  Usage  of  electronic  page-turners  and  portable 
document  formats  (PDFs)  with  limited  search  capabilities 
are  already  items  of  the  past.  Gulfstream  is  now  design¬ 
ing  aircraft  with  interactive  electronic  technical  manuals 
(IETM),  interactive  three-dimensional  models,  dynamic 
search  capabilities,  two-dimensional  illustrations,  and 
expanded  hyper-linking  (Figure  10). 

These  advancements  are  not  without  difficulties.  The 
proliferation  of  tablet  devices  has  made  it  necessary  to 


Figure  10.  The  Gulfstream  280  served  as  an 
impetus  for  change  in  technical  documentation. 
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deploy  publications  to  all  platforms  and  formats.  The  in¬ 
creased  partnership  in  aircraft  production  has  influenced 
sustainability  and  integration  of  data  from  multiple  sources. 

Paul  Mingler  -  GE  Aviation 

Mr.  Paul  Mingler,  Chief  Consulting  Engineer  of  Product 
Safety  for  GE  Aviation,  is  focused  on  product  safety  in  the 
design  of  the  engine,  has  been  involved  in  the  investigation 
of  all  accidents  involving  GE  commercial  engines,  and  has 
designed  a  number  of  proactive  safety  initiatives  to  improve 
propulsion  system  safety.  Mr.  Mingler  discussed  technical 
documentation  issues  in  terms  of  the  human  user  (e.g., 
technician,  engineer,  mechanic).  Over  the  last  30  years, 
we  have  seen  a  reduction  in  the  number  of  maintenance 
errors  attributable  to  machines.  Failure  to  follow  published 
technical  data  or  local  instructions  is  still  the  number  one 
cause  of  maintenance  mishaps  (Rankin,  2008). 

In  todays  organizations,  we  strive  to  be  learning  orga¬ 
nizations  that  conduct  analytics  (e.g.,  root-cause  investi¬ 
gations),  utilize  a  number  of  data  sources,  and  embody  a 
safety  culture.  This  has  led  to  an  improved  understanding 
of  the  maintenance  human  factors  lifecycle  -  design  for 
reliability  and  maintainability,  development  of  manuals  and 
procedures,  manufacturing  and  overhauling  the  aircraft, 
and  services  and  line  maintenance  throughout  the  aircraft  s 
life  span.  The  two  parts  of  the  process  that  seem  to  be  most 


relevant  to  the  issues  in  technical  documentation  are  “design 
for  reliability  and  maintainability”  and  “development  of 
manuals  and  procedures.”  In  designing  for  reliability  and 
maintainability,  it  is  critical  that  we  consider  the  design,  the 
engineering  process,  and  the  data  sources.  We  must  involve 
technicians  with  shop  and  line  maintenance  experiences, 
evaluate  design  reviews,  and  use  defensive  design  techniques, 
or  “Murphy-proof,”  whenever  possible. 

The  delivery  and  deployment  of  manuals  and  procedures 
must  consider  three  areas:  the  technical  publication  process, 
human  factors  aspects,  and  data  sources.  In  the  technical 
publication  process,  we  have  to  leverage  new  technologies, 
utilize  color-coded  warnings  and  cautions,  call-out  instruc¬ 
tions  in  sequence,  and  capitalize  on  the  benefits  of  pictures 
and  3D  graphics,  as  well  as  compliment  these  guides  with 
“kitted  parts.”  Manuals  and  procedures  developers  must 
consider  human  factor  aspects  such  as  intuitive  naviga¬ 
tion,  context  sensitivities,  and  training  aids  coupled  with 
manuals.  We  cannot  underestimate  the  human  factor  in  the 
technical  documentation  issues.  It  is  also  essential  that  we 
utilize  available  data  sources  to  improve  our  documentation. 
Mr.  Mingler  noted  they  use  a  combination  of  customer 
feedback,  revision  process  approach,  and  feedback  from 
training  schools  as  data  sources  in  the  development  and 
revision  of  manuals  and  procedures  (Figure  11). 
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Figure  11.  GE  Aviation  graphical  navigation  interface  with  color ;  3D  imaging ,  and  part  linkages. 


8 


•  Responds  to  events  that 
have  already  happened, 
such  as  incidents  and 
accidents 


Reactive 

(Past) 


Proactive 

(Present) 


Actively  seeks  the 
identification  of 
hazardous  conditions 
through  the  analysis  of 
the  organization's 
processes 


•  Analyzes  system 
processes  and 
environment  to  mdentify 
potential/futurc  problems 


Predictive 

(Future) 


Figure  12.  The  Safey  Management  System  process  that  should  be  initiated  at  the  OEM  level  for 
technical  documentation. 


As  the  aviation  maintenance  industry  moves  forward, 
we  must  consider  the  advancements  and  characteristics  of 
documentation  and  interface  design  experts,  Google®  and 
Apple®.  Change  is  happening  as  we  speak.  We  must  make 
sure  that  our  processes  and  equipment  can  accommodate 
new  technologies  and  be  applicable  across  four  genera¬ 
tions  in  the  workforce.  We  can  reduce  maintenance  error 
by  improving  our  technical  documentation  processes, 
development,  and  deployment. 

Lynn  Pierce  —  Federal  Aviation  Administration 
Mr.  Lynn  Pierce,  an  Aviation  Safety  Inspector  and 
Maintenance  Review  Board  chairman  for  the  Seattle  Air¬ 
craft  Evaluation  Group  (AEG),  is  known  for  his  thorough 
understanding  of  aircraft  technical  issues,  automation,  and 
aviation  business  practices.  Within  the  FAA,  Mr.  Pierce 
is  highly  involved  in  the  process  review  of  Instructions 
of  Continues  Airworthiness  (ICAs).  The  Seattle  AEG 
Airworthiness  Inspectors  in  his  group  reviews  1200  ICA 
packages,  with  some  projects  including  more  than  5000 
pages,  annually.  The  AEG  Airworthiness  Inspectors  review 
of  ICAs  conservatively  uses  33%  of  the  Inspector  total 
resources.  Nearly  all  of  the  ICA  packages  reviewed  receive 
a  tabletop  review,  with  little  or  no  witnessing  or  validation 
being  carried  out  regarding  the  ICA  packages.  Despite  the 
limited  review  resource  capabilities  of  the  AEG,  errors  are 
found  in  almost  every  project  reviewed.  This  is  of  great 
concern  when  considering  the  FAAs  current  policy  and 
position  regarding  Organization  Delegation  Authority 
(ODA).  With  the  implementation  of  ODA,  the  AEG 
has  found  itself  in  more  of  an  audit  function  of  ODA 
manual  and  processes  without  the  drill  down  and  review 
of  technical  data  once  accomplished  by  the  AEG.  Mr. 
Pierce  believes  we  must  begin  with  the  original  equipment 


manufacturer  to  start  to  resolve  the  documentation  and 
instructions  problems  we  see  today  throughout  the  avia¬ 
tion  industry.  Work  accomplished  with  inaccurate  data 
by  maintenance  personnel  can  result  in  unsafe  conditions 
and  put  the  mechanics  in  jeopardy  when  they  signed 
off  maintenance  with  incorrect  procedures.  Mr.  Pierce 
believes  we  must  utilize  Safety  Management  Systems 
(SMS)  processes  and  capitalize  on  reactive,  proactive, 
and  predictive  analysis  to  ensure  better  documentation 
and  more  accurate  data  (Figure  12). 

Since  Mr.  Pierce  authored  IP-44  (Issue  Paper)  a  few 
years  ago  for  the  International  Maintenance  Review 
Board  Policy  Board  (IMRBPB)  IP-44,  which  provides 
a  path  for  data  mining  and  statistic  analysis  when  look¬ 
ing  at  aircraft  data,  we  have  seen  Machine  Learning 
emerge  as  an  artificial  intelligence.  Machine  Learning 
uses  a  scientific  discipline  concerned  with  the  design 
and  development  of  algorithms  that  allow  computers  to 
evolve  behaviors  based  on  empirical  data,  such  as  from 
sensor  data  or  databases.  A  learner  can  take  advantage  of 
examples  (data)  to  capture  characteristics  of  interest  of 
their  unknown,  underlying  probability  distribution.  Data 
can  be  seen  as  examples  that  illustrate  relations  between 
observed  variables.  A  major  focus  of  machine  learning 
research  is  to  automatically  learn  to  recognize  complex 
patterns  and  make  intelligent  decisions  based  on  data. 
This  would  be  the  next  evolution  of  IP-44  and  how  we 
evaluate  and  use  data  in  the  future. 

Philippe  Barthas  —  Airbus  Company 

Mr.  Philippe  Barthas,  Senior  Director  of  Maintenance 
and  Repair  Technical  Data  for  the  Airbus  Company,  is 
responsible  for  delivering  and  supporting  the  maintenance 
and  repair  technical  data  for  Airbus  civil  aircraft.  Airbus 
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be  used  in  the  future . 

constantly  enhances  technical  data  (content  and  technol¬ 
ogy)  to  contribute  to  safety  in  maintenance  operations 
and  improve  the  efficiency  (usability)  of  maintenance 
documentation.  The  most  common  feedback  he  receives 
regarding  technical  data  usability  includes: 

•  task  card  or  steps  to  complete  the  task  are  unclear; 

•  tasks  involve  guesswork  or  trial  and  error; 

•  not  all  of  the  procedures  are  necessary; 

•  following  maintenance  manuals  slows  progress  making 
it  difficult  to  meet  deadlines; 


•  following  procedures  on  routine  tasks  is  an  inefficient 
use  of  time;  and 

•  some  procedural  steps  could  be  combined,  omitted, 
or  sequenced  differently  without  compromising  safety. 

To  address  these  issues,  Airbus  has  improved  the  qual¬ 
ity  of  technical  data  through  advanced  consultation,  en¬ 
hanced  data  deliverables,  and  sustained  customer  support 
(Figure  14).  To  enhance  consultation,  Airbus  has  created 
more  data  interoperability,  more  business-oriented  links 
between  documentation  modules,  and  quicker  access  to 
all  data  necessary  to  do  the  job. 

In  terms  of  data  deliverables,  Airbus  has  capitalized 
on  technological  advances  to  provide  dynamic  wiring 
functions  for  easier  and  faster  troubleshooting,  on-board 
integration  of  maintenance  data  and  the  on-board  infor¬ 
mation  terminals.  They  have  utilized  dynamic  displays 
of  troubleshooting  steps  to  show  “step-by-step”  displays 
for  ease  of  understanding  and  implemented  interactive 
graphics  and  3D  displays  and  enhanced  in-service  con¬ 
figuration  management.  He  also  noted  that  with  today’s 
technological  capabilities,  hyperlinks,  interactive  graphics, 
color  in  graphics,  warnings,  and  cautions  can  be  used 
much  more  extensively  than  in  the  past,  thus  contribut¬ 
ing  to  improved  clarity  and  understanding. 

To  also  improve  technical  data  operability,  Airbus  has 
worked  to  improve  the  process  to  establish  and  validate 
maintenance  procedures  to  ensure  they  meet  the  expecta¬ 
tions  of  the  end-users,  the  mechanics,  on  the  shop  floor. 

To  support  technical  data  operations,  Airbus  has 
implemented  websites  for  consultation,  downloading, 
reporting,  and  data  management  that  are  designed  to 


Technical  Data  is  a  significant  contributor  to 
safe  and  economical  aircraft  maintenance  operations 

AIRBUS  capitalizes  on  experience  to  constantly  ... 


O  Enhance  technical  content  for  operational  efficiency 

O  Develop  an  integrated  system  for  lean  and  seamless 
consultation 

O  Support  Technical  Data  operations  for  optimized 
administration  and  deployment 

Figure  14.  Airbus  objectives  to  continuously  enhance  technical  data  usability. 
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improve  data  availability  and  administration  of  infor¬ 
mation.  In  addition,  they  have  developed  a  Computer 
Assisted  Documentation  Education  Tutorial  System 
(C@DETS)  for  easy  airline  deployment  and  self-tutorial 
training.  They  have  also  initiated  a  Maintenance  Event 
Analysis  Panel  (MEAP) ,  which  reviews  all  in-service  events 
classified  as  being  significant  to  maintenance  operations, 
and  it  evaluates  contributing  factors  and  the  potential 
consequences  associated  with  errors.  The  output  of  the 


MEAP  is  used  to  provide  information  about  technical 
changes  in  maintenance  data  and  training  courses,  and 
to  communicate  with  customers. 

Discussants 

The  Honorable  John  Goglia  and  Dr.  Bill  Rankin  led 
the  group  discussion  at  the  end  of  the  session  on  creation 
and  delivery  of  technical  documents. 


1.5  WORKSHOP  -  DAY  2 


Figure  15.  Workshop  presentations  -  Day  2. 


1.5.1  Day  2:  Presentations 

The  final  presentations  (Figure  1 5)  conducted  during 
the  second  day  of  the  workshop  were  entitled  “Evidence- 
based  Practices.” 

1.5. 1.1  Session  3:  Evidence-Based  Practices 

Colin  Drury  —  Applied  Ergonomics  Inc. 

Dr.  Colin  Drury,  President  of  Applied  Ergonomics, 
Inc.  and  Distinguished  Professor  Emeritus  at  the  State 
University  of  New  York  (SUNY)  at  Buffalo,  made  the 
first  presentation.  SUNY-Buffalo  has  led  research  on  the 
application  of  human  factors  techniques  to  inspection 
and  maintenance  processes  for  more  than  20  years.  He 
opened  his  presentation  with  a  discussion  of  the  SHELL 
model  and  how  a  person  (liveware)  interacts  with  docu¬ 
mentation  (software),  hardware,  the  environment,  and 
other  people  (liveware)  to  accomplish  a  task.  Most  of  the 
technical  documentation  issues  originate  in  the  interac¬ 
tion  between  an  individual  and  software,  so  this  is  where 
improvements  are  needed. 


As  technical  documentation  is  developed,  the  goal  must 
be  error-free  performance.  Sometimes  we  become  sidetracked 
by  AMT/Inspector  preference,  minimizing  use  of  paper  or 
screens,  or  simplicity  of  the  documentation.  While  these  are 
not  bad,  we  have  to  focus  on  the  ultimate  goal  of  error-free 
performance  using  evidence-based  design.  We  must  under¬ 
stand  the  steps  an  AMT  must  follow  to  avoid  errors.  The 
technician  must  collect  the  correct  documentation,  read  it, 
comprehend  it,  and  perform  the  task  steps  correctly  from  the 
documentation.  The  four  types  of  errors  that  can  occur  are: 

1 .  Collect  wrong  documentation 

2.  Fail  to  read  documentation 

3.  Fail  to  comprehend  documentation,  and 

4.  Fail  to  execute  steps  correctly 

Dr.  Drury  reviewed  a  documentation  example  where  he 
worked  with  an  airline  partner  and  analyzed  data  on  actual 
errors  than  originated  from  using  a  hurriedly  developed  task 
card.  The  analysis  showed  unacceptable  error  rates  (1.5%  of 
responses,  20.8%  of  task  cards).  Task  card  errors  included 
issues  such  as  unexpected  placement  of  response  boxes,  illogi- 
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cal  instructions,  and  reference  to  different  pages.  None  of  the 
errors  occurred  when  human  factors  guidelines  were  met.  Dr. 
Drury  challenged  the  industry  to  fix  the  system  before  fixing 
the  liveware  (the  AMT) . 

Dr.  Drury  outlined  what  good  documentation  design 
should  look  like  (see  Figure  16).  In  particular,  it  should  fit  the 


task  to  the  user  and  be  designed  for  the  real  AMT/inspector, 
not  the  engineer  or  the  auditor.  There  are  a  number  of  tools 
currently  available  to  help  such  as  the  Documentation  Design 
Aid  available  at  https://hfskyway.faa.gov.  Research  has  shown 
that  application  of  the  design  principles  included  in  this  tool 
increases  comprehension  and  reduces  errors. 


THIS  SEQUENCE  INVOLVES  HAZARDOUS 
HYPERGCLIC  OPERATIONS.  PAD  CLEAR,  ORB ITER 


CONFIGURED  FOR  REMOTE  OPERATION  JiND 
PERSONNEL  IF  ON  STATION,  ATTIRED  IN  SCAPE 
SHALL  BE  VERIFIED  BEFORE  PROCEEDING. 

S0024  CONTROLS  THE  FOLLOWING  REQUIREMENTS: 

1.  CLEAR  PAD  TO  PERIMETER  SPACE 

2 .  VERIFY  ESSENTIAL  PERSONNEL  ATTIRED  IN 
SCAPE 

3.  INITIATE  RECORDING  OF  REQUIRED  CAMERAS 
ATIER  PERSONNEL  ARE  ON  STATION. 

CAUTION 

IF  THE  CRC6SFEED  AND  ENGINE  FEEDLINES  APE 


WARNING 
contains  action) 
steps 


Conditional 
steps  not  clear/ 


No  end  point  if\ 
condition  TRUE/ 


This  sequence  involves  hazardous  hype rgo lie 
operations.  Follow  this  procedure  from  S0024  foi^ 
safety: 


1.  Clear  pad  to  perimeter  space 

2.  Make  Sure  essential  personnel  are  attired  in 
scape 

3.  Start  recording  of  required  cameras  after 
personnel  are  on  station. 


IF  the  crossfeed  and  engine  feedlines  are 
softfilled 

AND  the  pressures  are  greater  than  70  psia, 
THEN  do  not  run  this  sequence  in  parallel  with  the 
flow  initiation  sequencer 
ELSE  ???????? 


The  following  sequence/ steps  contain  operations 
involving  Crit  1,  1R  and/or  IS  items. 


Figure  16.  Old  task  card  (top)  vs.  new  task  card  (bottom)  based  on  human  factors  design. 
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Overall,  Dr.  Drury  urged  industry  to  measure,  measure, 
measure.  Collected  data  can  be  used  to  improve  individual 
performance  and  overall  operations. 

Maggie  Ma  —  Boeing  Company 

Dr.  Maggie  Ma,  a  Systems  Engineer  for  maintenance 
human  factors  at  The  Boeing  Company,  works  closely 
with  airlines,  manufacturers,  maintenance  organizations, 
ground  service  providers,  and  international  regulatory 
agencies  to  apply  user-centered  design  methodology  to 
design,  develop,  and  test  a  variety  of  applications.  Dr. 
Ma  has  been  particularly  involved  in  examining  the  issue 
of  non-native  English  speakers  and  their  use  of  aviation 
maintenance  technical  documentation.  Currently  there 
are  two  categories  of  documentation  for  instructions  of 
continued  airworthiness  (ICAs):  maintenance  documents 
and  engineering  documents.  The  OEM  typically  produces 
ICAs  in  English,  but  the  maintenance  organization  may 
revise  the  ICAs  to  produce  bilingual  ICA  documents  or 
documents  in  their  technicians’  native  language.  This  tran¬ 
sition  of  information  from  the  OEM  to  the  maintenance 
organization  produces  an  opportunity  for  information  to 
get  “lost  in  translation.”  This  translation  issue  presents  an 
increasing  challenge  to  the  industry  as  outsourcing  has 
increased  from  37%  to  64%  (Dobbs,  2008). 

Dr.  Ma  reviewed  an  FAA  study  she  conducted  examin¬ 
ing  non-native  English  speakers  and  their  proneness  to 
increased  error  rates  (Figure  17).  In  this  study,  Dr.  Ma 
and  her  colleagues  measured  comprehension  of  task  cards 
(accuracy  and  time)  in  terms  of  three  factors:  task  card 
complexity,  type  of  document  English,  and  interventions. 


The  results  of  the  study  revealed  that  accuracy  was  good 
internationally,  reading  level  correlated  with  both  speed 
and  accuracy  across  continents,  full  and  partial  transla¬ 
tion  both  improved  accuracy  and  speed,  but  no  other 
interventions  helped  (e.g.,  glossary,  bilingual  coach). 
Ultimately,  the  researchers  concluded  that  language 
errors  do  occur,  but  they  can  be  reduced  or  eliminated 
through  training  and  translation.  Again,  measuring  er¬ 
rors  helps  reveal  those  strategies  that  did  work  and  those 
that  did  not.  Maintenance  organizations  must  be  aware 
that  time  pressure  compounds  errors  and  also  be  alert  to 
the  symptoms  of  flawed  communication  when  it  occurs. 

Guy  Minor  —  Federal  Aviation  Administration 

Mr.  Guy  Minor,  an  Aviation  Safety  Inspector  and  mem¬ 
ber  of  the  AFS-850  FAA  Safety  Team  Staff,  is  responsible 
for  educating  the  maintenance  industry  regarding  human 
error  and  just  culture.  Mr.  Minor  communicated  the 
capabilities  of  the  FAA  Safety  Team  to  address  technical 
documentation  problems  in  the  industry  (Figure  1 8) .  The 
FAA  Safety  Team  s  goal  is  to  improve  awareness  regard¬ 
ing  technical  documentation  issues  and  corresponding 
viable  user-centered  solutions.  The  FAA  Safety  Team 
utilizes  available  scientific  research  and  communicates 
with  industry  through  safety  meetings,  the  national  no¬ 
tice  system,  and  the  Wings  and  AMT  Awards  program. 

Discussant 

Mr.  Rayner  Hutchinson,  Vice  President  of  Quality 
and  Safety  for  AAR  CORP,  led  the  group  discussion  at 
the  end  of  the  session  on  evidence-based  practices. 


Improvements  with  T ranslation 


Figure  17.  Changes  in  accuracy  and  time  from  baseline  to  translated  conditions . 
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FAASafety.gov 


Markets  Safety  Meetings 

National  Notice  System 

Wings  and  AMT  Awards 

Hosts: 

Training 

Safety  Resources 
Directory 


Figure  18.  Summary  of  the  FAA  Safety  Team’s  support  of  improvements  in 
technical  documentation. 


Figure  19.  Workgroup  members  meet  to  identify  challenges  and 
short-  and  long-term  solutions. 


1.5.2  Day  2:  Small  group  event 

Following  the  last  workshop  presentation,  attendees 
volunteered  to  work  in  one  of  five  small  workgroups  to 
solve  different  technical  documentation  issues.  Attendees 
were  given  the  opportunity  to  serve  in  the  workgroup 
where  they  felt  their  expertise  would  be  most  relevant. 
Each  of  the  workgroups  consisted  of  four  to  seven  mem¬ 
bers.  The  workgroups  were  centered  around: 

•  document  quality, 

•  measurement, 

•  the  user/mechanic, 

•  the  government,  and 

•  industry  or  management. 


1. 5.2.1  Workgroup  task 

Workgroups  were  tasked  to  identify  five  challenges/ 
problems,  five  short-term  solutions,  and  five  long-term 
solutions  relevant  to  their  documentation  issue  and  to 
assign  a  spokesperson  to  present  the  workgroup  s  ideas 
to  the  large  group.  Attendees,  shown  in  Figure  19,  spent 
the  remainder  of  the  morning  and  worked  through  lunch 
on  the  task. 
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1. 5.2.2  Prioritization  of  workgroup  challenges  and 
solutions 

Following  detailed  presentations  of  identified  chal¬ 
lenges  and  proposed  solutions  by  each  workgroup,  at¬ 
tendees  were  asked  to  individually  rank  order  the  top  ten 
challenges  and  top  ten  solutions.  There  were  25  challenges 
and  52  solutions  (short-  and  long-term)  to  consider  in 
the  prioritization  task. 

1. 5.2.3  Evaluation  of  the  workshop 

Attendees  were  asked  to  complete  an  evaluation  form 
at  the  end  of  the  workshop. 

SECTION  2.0  WORKSHOP  RESULTS 

Analysis  of  the  prioritization  data  involved  assigning 
values  to  the  rankings.  The  first  choice  was  given  a  score 
of  1 0,  the  second  choice  was  given  a  score  of  9,  and  so  on. 

2.1  Prioritized  challenges 

A  list  of  25  challenges  emerged  from  the  five  work¬ 
ing  groups.  Table  1  lists  the  top  10  challenges,  of  the  25 
presented  by  the  workgroups,  in  the  order  of  priority 
based  on  overall  score. 

The  focus  of  the  workshop  was  on  solutions,  so  rather 
than  go  into  an  in-depth  review  of  the  challenges,  brief 
descriptions  of  the  top  five  challenges  will  be  presented 
in  subsections  2. 1.1 -2. 1.5. 

2.1.1  FAA  consistency 

There  is  insufficient  consistency  within  the  FAA,  spe¬ 
cifically,  the  interpretation  of  technical  data,  its  intent,  and 
its  alignment  with  regulatory  requirements.  Regulatory 
expectations  and  requirements  continue  to  change,  and 

Table  1.  Prioritized  List  of  Challenges 

Priority  Challenge 

1  FAA  consistency  (DQ) 

2  Content  accuracy  (U) 

3  Culture  of  noncompliance  with  documentation  (I) 

4  Lack  of  appropriate  business  case  for  document  issue  improvement  (M) 

5  Industry  standards  (DQ) 

6  Clarity  of  regulatory  requirements  (G) 

7  Content  availability  (U) 

8  OEM  data  quality  and  usability  (DQ) 

9  No  quick  and  valid  measure  of  document  quality  (M) 

10  Balance  production/safety,  compliance,  quality  (I) 

Workgroup  designators:  DQ-document  quality;  M-measurement;  U-user/mechanic;  G-government(G); 
l-industry  or  management  (I). 


the  understanding  of  the  linkage  between  the  intent,  the 
requirements,  and  the  technical  documentation  can  be 
difficult  to  maintain.  The  AEG  s  oversight  of  technical 
publications  was  identified  as  the  number  one  challenge 
facing  the  operators.  One  issue  discussed  enforcement 
and  interpretation  of  requirements  that  seemed  to  be 
regional,  or  even  specific  to  an  inspector,  rather  than 
general  and  consistent  in  all  offices.  For  instance,  an 
aircraft  (or  component)  built  in  Kansas  might  not  have 
the  same  documentation  requirements  and  oversight  as 
one  built  in  Georgia. 

2.1.2  Content  accuracy 

Accuracy  of  technical  documentation  is  paramount  in 
assuring  that  proper  maintenance  actions  are  performed 
in  service.  However,  integration  of  OEMs,  Supplemental 
Type  Certificate  (STC),  supplier  data,  and  operator  data 
within  the  operations  environment  is  also  key  in  deliver¬ 
ing  on  the  promise  of  accurate  data  (i.e.,  the  right  data, 
at  the  right  time,  in  a  usable  format  for  the  environment 
at  hand).  The  opportunity  to  address  data  quality,  from 
the  using  community’s  perspective,  is  in  the  integration 
of  data  delivery,  user  feedback,  and  data  updates  that 
address  the  real-time  needs  of  the  user  groups.  Today, 
organizations  have  a  difficult  time  applying  the  right 
resources,  with  the  right  knowledge,  skills,  and/or  system 
solutions  to  address  the  day-to-day  issues  created  by  the 
lack  of  technical  data  integration. 

2.1.3  Culture  of  noncompliance  with  documentation 

There  are  many  organizational  norms  regarding  docu¬ 
mentation  use.  A  norm  that  gives  tacit  approval  to  ignore 
technical  publications  for  some  maintenance  tasks  will 
have  negative  consequences.  This  type  of  norm,  perhaps 
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due  to  the  perception  of  timesavings,  tends  to  spread  to 
an  increasing  number  of  tasks  until  an  incident  occurs 
or  additional  maintenance  costs  are  realized.  Like  a  bad 
habit,  organizational  norms  can  present  many  challenges 
to  changing  cultures,  processes,  and  tools. 

2. 1 .4  Lack  of  appropriate  business  case  for  docu¬ 
ment  issue  improvement 

The  case  can  be  difficult  to  make  for  investing  in 
technical  information  improvements.  Large  and  small 
category  aircraft  OEMS  know  that  adherence  to  technical 
documentation  regulations  is  good  business.  However, 
integration  and  management  of  all  technical  documenta¬ 
tion  requirements  remains  a  shared  responsibility.  The 
business  case  justification  for  investing  in  improved 
technical  data  management  requires  improved  linkage 
between  the  cost  of  compliance  and  maintaining  a  safe 
and  efficient  work  place.  Even  though  there  are  known 
costs  associated  with  maintenance  errors  (delays,  damage, 
personal  injury),  it  can  be  difficult  to  correlate  savings 
from  cost  avoidance  items  like  reduced  errors  through 
improved  data  and  data  management.  Effective  data 
management  (i.e.,  organizational  culture  where  “zero” 
violations  is  the  goal)  and  a  robust  data  management 
investment  plan  holds  potential  for  all  participants  to 
benefit  —  a  win-win  scenario  for  all. 


2.1.5  Industry  standards 

Section  2.1.1  noted  the  inconsistent  application  of 
FAA  standards  regarding  the  use,  interpretation,  and 
compliance  expectations  of  technical  documentation. 
The  case  is  the  same  for  industry  standards.  Current  Air¬ 
lines  for  America  (A4A;  also  known  as  Airline  Transport 
Association,  or  ATA)  documents  capitalize  on  Standard 
Generalized  Markup  Language  (SGML).  The  industry 
is  moving  forward  with  S1000D  specifications  that  use 
Extensible  Markup  Language  (XML)  coding  that  is  con¬ 
ducive  to  relational  databases.  There  must  be  a  push  to 
move  aggressively  on  such  standards,  and  to  let  these  types 
of  standards  to  become  the  guidelines  for  FAA  approval. 

2.2  Prioritized  Solutions 

The  five  working  groups  (as  noted  in  the  previous  sec¬ 
tion)  generated  a  list  of  52  solutions  or  action  items.  The 
top  ten  solutions,  shown  in  Table  2,  are  the  workshops 
action  items  for  addressing  technical  documentation 
issues.  Note  that  there  is  not  a  direct  mapping  between 
the  two  prioritized  lists,  since  attendees  ranked  solutions 
and  challenges  separately. 

The  following  subsections,  2.2. 1-2.2.10,  elaborate  on 
each  action  item.  Workshop  discussions  were  the  basis 
for  the  final  recommendations,  with  some  presented  in 
terms  of  industry,  government,  and  Individual  actions. 


Table  2.  Top  Ten  Action  Items  to  Address  Documentation  Issues 


Priority 

Action  Item 

1 

Quantify  financial  loss  related  to  documentation  issues  (M) 

2 

Develop/apply  methods  for  evaluating  the  quality  of  technical  documentation  (M) 

3 

Leverage  voluntary  reporting  to  identify  specific  problems  with  documentation  (1) 

4 

Improve/create  guidance  for  FAA  personnel  working  documentation  issues, 
especially  Instructions  for  Continued  Airworthiness  (G) 

5 

Expand  incident  investigation  to  identify  details  associated  with  documentation 
issues  (M) 

6 

Improve  integration  and  linkage  of  content  across  maintenance  documents  - 
maintenance  manuals,  task  cards,  and  illustrated  parts  catalogs  (U) 

7 

Delegate  approval  from  FAA  to  industry  using  established  Organization 
Designation  Authorization  (DQ) 

8 

Improve  usability  of  manual  format,  accessibility  of  manual,  and  training  on 
manual  use  (U) 

9 

Initiate  industry  mandate  requiring  users  to  address  known  documentation 
issues.  This  applies  to  all  levels  of  the  organization  (1) 

10 

Improve  coordination  of  document  professionals  from  industry  segments  and 
government  (1) 

Workgroup  designators:  DQ-document  quality;  M-measurement;  U-user/mechanic;  G-government(G); 
l-industry  or  management  (I). 
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2.2.1  Quantify  financial  loss  related  to  documenta¬ 
tion  issues 

The  highest  priority  action  item  is  to  quantify  the 
costs  associated  with  documentation  issues.  Airlines  do 
not  know  the  cost  of  errors  associated  with  legacy  docu¬ 
mentation  systems,  and  they  are  currently  unlikely  to  ap¬ 
preciate  the  potential  cost  savings  from  new  systems.  This 
is  quite  natural:  People  underestimate  the  frequency  and 
cost  of  rare  events,  such  as  errors  that  propagate  through 
the  system  to  become  incidents  or  accidents.  Workshop 
participants  felt  that  if  industry  really  understood  the 
costs  associated  with  poor  documentation,  then  they 
would  invest  in  improving  it. 

Currently  there  are  limited  regulatory  reasons  that 
require  aircraft  or  component  manufacturers  to  improve 
the  usability  and  effectiveness  of  their  documents.  Of 
course,  there  is  market  incentive  for  these  parties  to  in¬ 
vest  in  radically  new  documentation  systems  that  make 
their  aircraft  more  maintainable.  However,  aircraft  and 
systems  are  marketed  primarily  on  passenger  comfort 
and  acceptance,  fuel  efficiency,  extended  inspection  and 
maintenance  cycles,  environmental  impact,  and  other 
tangible  factors  to  which  a  finance  department  can  assign 
value.  There  is  less  marketing  appeal  in  the  efficiency  and 
safety  gains  that  improved  technical  documentation  may 
provide.  New  generation  aircraft  have  improved  on-board 
maintenance  documentation,  portable  maintenance  aids, 
and  documentation  to  be  more  compatible  with  portable 
computer  systems  and  todays  airline  operational  reality. 
Such  systems  are  a  step  in  the  right  direction  but  are 
seldom  the  most  appealing  factor  that  separates  one  new 
aircraft  from  the  next. 

A  majority  of  the  U.S.  airline  fleet  average  about  12.5 
years  old.  That  means  there  is  a  legacy  documentation 
system  that  is  not  as  advanced  as  a  new  Airbus,  Boeing, 
Bombardier,  or  Embraer.  It  would  take  very  significant 
investment  to  review  and  modify  the  legacy  documenta¬ 
tion  system.  It  would  be  difficult  to  quantify  the  safety  and 
financial  benefits  of  a  large  investment  in  new  technical 
documentation  systems  since  it  would  require  baseline 
data  on  the  cost  of  errors  associated  with  the  old  system. 
However,  legacy  aircraft  will  remain  in  the  U.S.  and 
world  fleets  for  many  years,  with  the  latent  errors  of  poor 
documentation  remaining  an  unaccounted  safety  risk. 

The  bottom  line  is  that,  to  date,  there  has  not  been 
a  good  answer  regarding  the  financial  -  much  less  the 
safety  impact  —  of  technical  documentation.  Quantifying 
the  cost  of  documentation  issues  may  incentivize  aircraft 
operators  and  suppliers  to  address  technical  documenta¬ 
tion  problems  and  better  inform  their  solutions.  Accom¬ 
plishing  Action  Item  1  means  that  an  operator,  MRO, 
or  OEM  assign  costs  to  maintenance  errors  attributable 


to  documentation  issues  (e.g.,  reworks,  quality  escapes, 
aircraft  damage,  personal  injury).  They  could  also  assign 
safety  impact. 

The  government  should  consider  authoring  an  Advi¬ 
sory  Circular  or  an  information  document  describing  a 
Cost  of  Quality  (CoQ)  program  and  recommending  that 
industry  use  it.  Participation  would  be  voluntary;  however, 
a  standardized  CoQprogram,  such  as  that  recommended 
by  the  American  Society  for  Quality  (ASQ) ,  could  provide 
a  standardized  method  to  collect,  cost,  report,  and  analyze 
quality  escapes  within  the  industry.  Given  that  the  total 
CoQ  can  often  be  20-30%  of  sales,  this  initiative  could 
have  huge  ramifications  (Westcott,  2006). 

When  a  documentation  issue  arises,  establish  the  costs 
associated  with  time,  error,  rework,  etc.  Cost  data  should 
include  obvious  measures  associated  with  documentation 
issues  (e.g.,  not  available,  not  up-to-date,  difficult  to  use) 
that  contribute  to  aircraft  damage,  delays,  and  in-flight 
returns,  as  well  as  other  performance  indicators.  The  main 
cost  is  that  of  an  accident,  an  extremely  rare  event,  but  one 
that  can  bankrupt  a  carrier  (e.g.,  Value  Jet,  PanAm).  Such 
events  do  not  fit  well  with  current  accounting  practice, 
nor  does  it  mean  that  we  can  ignore  them. 

Industry  Actions 

•  Develop  a  formal,  standardized  method  to  capture 
quality  escapes. 

•  Encourage  use  of  voluntary  reporting  systems  to  note 
documentation  challenges. 

•  Design  mechanisms  to  collect  all  documentation  er¬ 
rors/challenges  to  complete  the  picture  depicted  by 
the  voluntarily  reported  data. 

•  Assign  cost  values  to  errors,  delays,  etc.,  from  docu¬ 
mentation  problems. 

•  Collect  and  evaluate  the  number  of  FAA  administrative 
actions  resulting  from  documentation  issues. 

Government  Actions 

•  Use  Aviation  Safety  Information  Analysis  and  Shar¬ 
ing  (ASIAS)  system  to  identify  documentation  issues 
annually. 

•  Help  industry  assign  safety  and  financial  cost  associ¬ 
ated  with  incident  reports. 

•  Encourage  voluntary  reporting  and  special  consider¬ 
ation  for  documentation-related  reports. 

•  Train  ASAP  participants  in  industry  and  in  FAA  to 
emphasize  documentation  issues. 

•  Take  administrative  action  on  airlines  and  manufactur¬ 
ers  that  produce  and  use  suboptimal  documentation. 

•  Commit  to  and  fund  applied  R&D  to  provide  guid¬ 
ance  and  training  to  those  who  produce  and  use 
documentation. 
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•  Consider  authoring  an  Advisory  Circular  or  informa¬ 
tion  document  describing  recommending  use  of  a 
CoQ  program  within  industry. 

Individual  Actions 

•  If  the  documentation  is  unavailable  or  incorrect,  then 
engage  the  engineering  and  quality  assurance  depart¬ 
ments  to  resolve  the  problem. 

•  Report  ALL  suboptimal  work  card  or  manufacturer’s 
instruction  to  your  management  and  to  FAA. 

•  Use  ASAP  and  other  voluntary  reporting  systems  to 
highlight  the  documentation  issues. 

2.2.2  Develop/apply  methods  for  evaluating  quality 
of  technical  documentation 

What  characteristics  define  good  documentation?  We 
know  that  good  documentation  aids  users  in  perform¬ 
ing  critical  tasks  in  maintenance  and  inspection  because 
it  reduces  error  due  to  misunderstanding  and  prevents 
users  from  ignoring  paperwork  and  failing  to  perform 
tasks  according  to  the  instructions. 

Industry  has  made  some  maj  or  changes  in  how  technical 
documentation  is  delivered  to  the  user.  Although  there 
has  been  an  increase  in  the  diversity  of  delivery  media, 
there  does  not  appear  to  be  a  commensurate  increase  in 
the  quality  of  the  documentation  delivered. 

The  major  airframe  manufacturers  have  studied  docu¬ 
mentation  quality  with  respect  to  usability  (e.g.,  using 
Aerospace  and  Defence  Industries  Associations  of  Europe 
SimplifiedTechnical  English) ,  but  the  engineers  who  write 
technical  directions  are  not  necessarily  well-versed  in  the 
science  of  writing  procedures— neither  are  the  FAA  AEG 
inspectors  who  write  evaluation  guidelines  or  the  FAA 
inspectors  who  evaluate  documents  in  the  field.  These 
groups  of  users  need  a  valid,  reliable,  straightforward  way 
to  produce  and  evaluate  quality  documentation.  There  is 
a  corpus  of  human  factors  literature  available,  some  of  it 
funded  by  the  FAA,  on  designing  paper  and  computer- 
based  documentation  (i.e.,  Patel,  Drury,  &  Lofgren, 
1994;  Drury,  1998;  Drury  &  Sarac,  1997;  Drury,  Patel, 
&  Prabhu,  2000)  and  tools  for  auditing  human-system 
mismatches  in  maintenance  (Koli,  Chervak,  &  Drury, 
1999). 

The  methodology  for  evaluating  documentation  qual¬ 
ity  should  be  applied  to  new  media  and  recent  documenta¬ 
tion  designs  discussed  in  the  workshop  that  incorporate 
3D  animated  graphics,  in  particular.  The  use  of  sound 
design  principles  for  documentation  as  the  basis  for  an 
audit  checklist  would  help  ensure  “content  validity”  of 
the  final  checklist.  In  aviation  maintenance,  a  current 
example  is  the  checklist  for  each  NDI  (nondestructive 
inspection)  technique  developed  as  part  of  the  Handbook 
of  System  Reliability  in  Airframe  and  Engine  Inspection 


(Drury,  2005).  These  design  approaches  and  evaluation 
tools  might  also  support  maintenance  training. 

Furthermore,  development  of  a  document  evaluation 
methodology/ tool  will  require  valid  definitions  for  all  as¬ 
pects  of  quality ,  such  as  the  design  guidance  embedded  in 
existing  document  design  tools.  As  part  of  the  development 
process,  the  methodology/tool  needs  to  be  empirically 
tested  to  ensure  that  its  output  is  reliable  (i.e.,  it  produces 
similar  results  when  used  by  different  evaluators  or  used 
by  the  same  evaluator  across  different  types  of  technical 
documentation)  and  valid  (i.e.,  the  results  correlate  well 
with  an  objective  criterion  such  as  comprehension  error 
rates  of  documentation  users).  The  tool  also  needs  to  be 
designed  based  on  user  requirements — easy  and  quick, 
and  handles  multiple  forms  of  technical  documentation. 

Testing  reliability,  validity,  and  acceptability  of  a 
methodology/ tool  for  assessing  document  quality  re¬ 
quires  actual  users  evaluating  known  levels  of  document 
quality.  Reuse  of  paper  and  computer-based  documents 
from  prior  studies  where  comprehension  error  rates  have 
already  been  measured  could  reduce  the  size  of  the  ef¬ 
fort.  However,  for  newer  media,  such  as  animated  3D 
graphics,  new  approaches  and  additional  materials  may 
need  to  be  developed  and  tested. 

Industry  Actions 

•  Commit  to  use  existing  guidance  documentation  for 
review  and  revision  of  technical  documentation  that 
is  identified  as  problematic. 

•  Use  voluntary  reporting  systems  to  note  documenta¬ 
tion  problems. 

•  Be  willing  to  collect  and  evaluate  the  cost  of  the  errors 
associated  with  poor  documentation. 

Government  Actions 

•  Fund  research  and  development  to  reintroduce  proper 
documentation  practices. 

•  Create  a  training  course  for  documentation  develop¬ 
ment  and  evaluation. 

•  Analyze  the  voluntary  reporting  data  to  identify  the 
common  characteristics  for  poor  documentation. 

•  Renew  Advisory  Circulars  and  other  relevant  guidance 
material  for  industry  and  for  inspectors. 

Individual  Actions 

•  If  the  documentation  is  unavailable  or  incorrect,  then 
engage  the  engineering  and  quality  assurance  depart¬ 
ments  to  resolve  the  problem. 

•  Report  ALL  suboptimal  work  card  or  manufacturer  s 
instruction  to  your  management  and  to  FAA. 

•  Use  ASAP  and  other  voluntary  reporting  systems  to 
highlight  the  documentation  concerns. 
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2.2.3  Leverage  voluntary  reporting  to  identify 
specific  problems  with  documentation 

To  facilitate  company  responsiveness  in  resolving 
documentation  issues,  the  number  3  solution  was  use  of  a 
FAA / company  supported,  non-punitive  reporting  process, 
with  the  company  taking  responsibility  for  informing  the 
workforce  of  corrective  actions  to  incentivize  its  use. 

Recommendations  regarding  Action  Item  3  are  some¬ 
what  related  to  Action  Item  5,  as  they  both  deal  with  im¬ 
proved  reporting  and  collecting  of  technical  documentation 
deficiencies.  Voluntary  disclosure  programs  are  increasingly 
popular  as  SMS  evolve.  This  is  true  for  the  air  carrier,  general 
aviation,  and  MRO  environments.  Example  programs  in 
the  U.S.  include  the  FAA  ASAP,  the  new  FAA  Maintenance 
and  Ramp  Line  Operations  Safety  Assessment  (LOSA),  and 
the  long-standing  NASA  ASRS.  These  programs  permit 
personnel  to  report  errors  anonymously,  usually  avoiding 
any  FAA  civil  action.  For  instance,  a  mechanic  could  use 
existing  government-sponsored  programs  to  report  instances 
of  deviations  from  the  14  CFR  Part  43  or  Part  121  rule 
applied  to  technical  publications. 

The  FAA  should  create  materials  to  help  identify  the 
type  of  information  and  level  of  detail  needed  in  a  report 
to  communicate  (clear,  correct,  complete,  and  concise) 
a  documentation  issue.  The  information  should  allow 
categorization  of  the  reported  error  -  to  determine  if  it 
was  attributable  to  performance,  task  conditions,  accepted 
practices  within  the  organization,  and/or  quality  of  the 
documentation  (e.g.,  design,  currency,  and  availability). 

There  are  certain  tasks  that  industry  does  well  and  other 
tasks  that  are  more  appropriate  for  government.  When  the 
task  is  generic  and  all  in  the  industry  share  the  benefits, 
then  it  should  be  a  government  task.  Of  course,  it  is  best 
when  there  are  industry  participants  on  the  government 
development  team. 

Industry  Actions 

•  Change  use  of  voluntary  reporting  systems  to  empha¬ 
size  documentation  issues  (FAA  R&D  can  support 
this  effort). 

•  Encourage  voluntary  reports  about  documentation  even 
in  advance  of  an  event. 

•  Pay  employees  for  high  value  voluntary  reports  on 
documentation  problems. 

Government  Actions 

•  Use  ASIAS  system  to  identify  documentation  issues 
annually.  Transition  from  data  collection  to  informa¬ 
tion  reporting. 

•  Encourage  voluntary  reporting  and  special  consideration 
for  documentation-related  reports. 

•  Train  ASAP  participants  in  industry  and  in  FAA  to 
emphasize  documentation  issues. 


•  Create  checklists  for  documentation  evaluation  (Applied 
R&D  effort). 

Individual  Actions 

•  Report  ALL  suboptimal  work  card  or  manufacturers 
instruction  to  your  management  and  to  FAA. 

•  Use  ASAP  and  other  voluntary  reporting  systems  to 
highlight  the  documentation  challenges. 

•  If  ASAP  does  not  offer  a  means  to  report  documenta¬ 
tion  issues,  then  report  the  situation  to  ASAP  manager. 

2.2.4  Improve/create  guidance  for  FAA  personnel 
working  documentation  issues,  especially  Instruc¬ 
tions  for  Continued  Airworthiness  (ICA) 

Action  item  4  is  inherently  associated  with  Action  Item 
2.  The  issue  focuses  on  FAA  personnel  who  review  the  ICA 
documents,  which  include  maintenance  manuals,  illustrated 
parts  catalogs,  and  fault  isolation/ troubleshooting  manuals. 
These  are  critical  documents  in  a  manufacturers  approved 
documentation  package 

Tools  and  training  should  be  developed  and  applied  to 
empower  government  and  industry  personnel  who  create 
and  validate  the  technical  instructions,  presumably  based 
on  the  evaluation  methodology  developed  in  Action  Item 
2.  Industry  and  government  workshops  on  topics  like  A4A 
S1000D  would  be  an  excellent  start.  Accomplishing  this 
involves  making  it  a  requirement  and  funding  it  accordingly. 

Industry  Actions 

•  Commit  to  use  existing  guidance  for  review  and  revision 
of  technical  documentation 

•  Use  voluntary  reporting  systems  to  note  documenta¬ 
tion  challenges. 

•  Strive  for  standardization  among  all  documents. 

Government  Actions 

•  Renew  Advisory  Circulars  and  other  relevant  guidance 
material  for  industry  and  for  inspectors. 

•  Simplify  oversight  guidelines. 

•  Assign  responsible  parties  to  work  the  issue. 

•  Fund  research  and  development  to  reintroduce  proper 
documentation  practices. 

•  Create  training  course  for  documentation  development 
and  evaluation. 

•  Analyze  the  voluntary  reporting  data  to  determine  the 
common  characteristics  for  poor  documentation. 

•  Create  special  training  course  for  Aviation  Safety  Inspec¬ 
tors  who  will  work  in  AEG. 

Individual  Actions 

•  Report  ALL  suboptimal  work  card  or  manufacturers 
instruction  to  your  management  and  to  FAA. 
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•  When  something  is  wrong,  file  a  report  in  the  Aviation 
Safety  Reporting  System  and  cite  a  violation  of  14  CFR 
Part  43  13(a). 

2.2.5  Expand  incident  investigation  to  identify  de¬ 
tails  associated  with  documentation  issues 

Current  requirements  for  incident/ accident  investigations 
must  require  a  complete  root  cause  analysis  (RCA)  when  it 
appears  “a  procedure  was  not  followed.”  The  cause  may  be  a 
maintenance  error,  where  a  well-documented  procedure  was 
not  properly  executed.  Or  the  cause  may  be  a  document- 
related  issue  where  the  instructor  was  not  clear,  complete, 
or  correct.  Although  sufficient  for  an  investigation,  the 
categories  are  too  broad  from  the  standpoint  of  actionable 
change  to  mitigate  future  risk.  As  well,  existing  reporting 
systems  (mentioned  in  section  2.3)  may  contain  information 
relevant  to  an  investigation,  but  it  often  does  not  explicitly 
state  what  the  precursor  was  to  a  maintenance  error. 

The  FAA  should  commit  to  a  research  and  develop¬ 
ment  effort  focused  on  improving  outcomes  of  formal 
and  informal  investigations  of  maintenance  errors  and 
document-related  issues  by  improving  the  data  quality 
and  developing  analysis  tools,  e.g.,  based  on  the  tools 
developed  in  Action  Item  2.  FAA  researchers  could  assist 
in  determining  what  the  “right”  questions  are  for  a  report¬ 
ing  system  during  a  formal  investigation.  They  could  also 
create,  test,  and  implement  an  analysis  tool,  similar  to  that 
created  for  fatigue  risk  assessment,  to  support  analyzing 
documentation-related  information  output  from  existing 
reporting  systems.  This  line  of  research  would  apply  to 
the  recommendation  in  section  2.3  for  FAA/ company- 
supported  internal,  voluntary  reporting  programs. 

Industry  Actions 

•  Change  current  event  and  accident  investigation  forms 
to  have  a  section  dedicated  to  the  use  of  technical  docu¬ 
mentation.  If  technical  documentation  is  suspect,  then 
ensure  sufficient  root  cause  analyses. 

•  Create  internal  procedures  for  immediate  correction  of 
technical  documentation  any  time  it  may  be  a  small  (or 
large)  contributing  factor  to  a  safety  event. 

Government  Actions 

•  Create  procedures  and  training  to  help  event  investigators 
recognize  the  manner  in  which  technical  documenta¬ 
tion  may  have  been  an  contributing  factors  to  an  event. 

•  Ensure  that  all  FAA  accident  investigations  that  identify 
procedures  or  documentation  as  contributing  factors  also 
ensure  that  a  proper  root  cause  analysis  is  conducted. 

Individual  Actions 

•  Report  all  detailed  information  about  documentation 
issues  that  contributed  to  an  error/event/accident. 


2.2.6  Improve  integration  and  linkage  of  content 
across  maintenance  documents  —  maintenance 
manuals,  task  cards,  and  illustrated  parts  catalogs 

Today,  line  and  heavy  maintenance  are  accomplished 
more  rapidly  and  in  faster-paced  environments.  The  main¬ 
tenance  data  and  the  manuals  that  contain  the  relevant 
information  needed  to  perform  the  required  tasks  include 
an  ever-increasing  amount  of  information.  Accessing  this 
data  efficiently  and  accurately  has  become  paramount 
for  the  mechanic,  engineering,  manufacturing,  quality, 
and  management  staffs  to  perform  their  work.  Access  to 
the  right  data,  in  real-time,  on-demand,  and  in  the  work 
environment  is  more  important  today  than  ever. 

2.2.6. 1  Providing  the  right  access 

To  provide  the  right  access,  accurate  links  between 
the  required  technical  information  (i.e.,  maintenance 
documents,  engineering  data,  quality  control  data,  and 
maintenance  records)  is  critical  in  supporting  repeatable 
maintenance  actions.  OEMs,  third  party  providers,  and 
maintenance  data  solution  providers  (along  with  industry) 
should  look  to  provide  the  links  between  datasets  that  al¬ 
low  for  quicker,  more  accurate,  and  repeatable  processes 
for  accomplishing  the  required  maintenance  actions  in  the 
line  and  heavy  maintenance  environments. 

2.2.6.2  Providing  the  right  tools 

Delivery  tools  should  be  developed  to  allow  for  the  single 
point  of  use  of  all  relevant  maintenance,  engineering,  and 
quality  data  in  the  accomplishment  of  required  in-service 
maintenance.  This  includes  access  to  the  required  infor¬ 
mation  at  line  and  heavy  stations.  To  go  further,  todays 
datasets  (i.e.,  OEM,  Supplier,  and  Airlines)  should  address 
the  emerging  requirements  for  real-time,  mobile,  and 
point-of-use  access  to  data  that  the  business  environment 
demands  in  all  industries. 

2.2.6.3  Providing  the  right  solutions 

Finally,  integration  of  data  management  tools  is  needed  to 
allow  for  the  inter-linking  of  required  datasets  (i.e.,  OEM, 
Airline,  Third  Party,  and  supplier) .  Integration  of  the  data 
through  database  management  offers  the  opportunity  to 
improve  access,  understanding,  and  oversight  of  mainte¬ 
nance  operations  for  all  involved  (e.g.,  OEM,  airlines,  and 
regulatory  agencies).  Integrated  solutions  for  managing 
maintenance  activity  will  enhance  maintenance  execution, 
management,  and  oversight  not  achievable  by  other  means. 

Industry  Actions 

•  Manufacturers  should  take  the  lead  and  work  with 

FAA  AEG  to  standardize  integration/connectivity  of 

documents. 

•  Evaluate  the  utility  of  new  technology. 
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•  Seek  employee  and  customer  guidance  on  this  topic. 

•  Obtain  corporate  commitment  to  this  effort. 

•  Consider  developing  a  single  portal  where  contents  are 
similarly  formatted  and  easily  accessible. 

Government  Actions 

•  Coordinate  with  industry. 

Individual  Actions 

•  Give  ideas  for  improved  access  to  data. 

2.2.7  Delegate  approval  from  FAA  to  industry  using 
established  Organization  Designation  Authorization 
(ODA) 

The  ODA  program  empowers  private  companies  to 
take  the  formal  responsibility  for  approving  their  own 
documentation  without  extensive  oversight  from  the  FAA. 
With  ODA  the  FAA  oversees  the  process  of  approval  but 
does  not  review  the  details  of  a  company’s  product  technical 
documentation.  The  process  relieves  some  of  the  detailed 
FAA  workload  and  delay  in  obtaining  an  approval.  The 
workshop  attendees  recommend  that  the  FAA  more  broadly 
apply  ODA  for  this  purpose. 

Industry  Actions 

•  Propose  to  FAA  better  ways  to  delegate  authority. 

Government  Actions 

•  Capitalize  on  ODA  to  a  greater  extent. 

•  Create  the  guidance  materials  to  empower  industry 
for  self- approval  and  to  help  FAA  inspectors  to  better 
oversee  the  Designated  Approval  process. 

•  Take  actions  to  improve  documentation  guidance  mate¬ 
rial.  Assign  a  responsible  person  and  organization.  Set 
timetables  for  production  and  adhere  to  them. 

2.2.8  Improve  usability  of  manual  format, 
accessibility  of  manual,  and  training  on  manual  use 

Accomplishing  Action  Item  8  requires  additional  pub¬ 
lished  guidance  for  all  document  creators  and  reviewers. 
Workshop  attendees  noted  that  some  instructions  are  sim¬ 
ply  “unusable,”  and  procedures  have  not  been  sufficiently 
validated  by  the  manufacturer  engineers  who  prepare  the 
instructions.  If  so,  a  review  of  the  validation  process  and 
usability  assessment  is  needed  at  the  manufacturer  or  the 
operator  level,  when  specific  maintenance  procedures  are 
written  and  job  cards  are  created.  Regardless,  there  is  an 
opportunity  for  improvement. 

Industry  Actions 

•  Change  voluntary  reporting  systems  to  emphasize 
documentation  challenges  (FAA  R&D  can  support 
this  effort). 


•  Take  an  active  lead  in  building  documents  that  easily 
plug  into  more  useful  systems  of  delivery. 

•  Pay  employees  for  high-value  voluntary  reports  on 
problematic  documentation  (this  is  an  investment  for 
change). 

•  Set  specific  goals  for  revisions  of  the  problematic  docu¬ 
ments. 

Government  Actions 

•  Train  ASAP  participants  in  industry  and  in  FAA  to 
emphasize  documentation  issues. 

•  Create  checklists  for  documentation  development  and 
evaluation  (applied  R&D  effort). 

•  Update  guidance  material  and  applied  R&D  results. 

Individual  Actions 

•  Report  ALL  suboptimal  work  card  or  manufacturer’s 
instruction  to  your  management  and  to  FAA. 

•  Push  for  change  as  individual  maintenance  personnel 
by  reporting  the  problem/issue  to  the  engineering  and 
quality  assurance  departments. 

•  Complete  an  ASRS  report  to  document  problem. 

•  OEM  and  operator  authors  need  to  have  basic  knowledge 
of  usability  and  how  it  applies  to  technical  documenta¬ 
tion  design  and  authoring. 

2.2.9  Initiate  industry  mandate  requiring  users  to 
address  known  documentation  issues 

Requests  from  users  for  manual  revisions  are  often 
lumped  with  other  company  requests  and  not  addressed 
in  a  timely  manner.  When  users  do  not  see  timely  results 
from  manual  revision  requests,  they  frequently  give  up  and 
create  “workarounds”  to  complete  tasks,  ignoring  deficien¬ 
cies  in  the  manuals  until  a  problem  occurs.  The  operators 
and  manufacturers  must  make  a  renewed  commitment  to 
expedite  the  process  to  address  issues  with  documentation 
and  consider  a  minimum  time  limit  that  any  documenta¬ 
tion  issue  remains  open. 

Industry  Actions 

•  Revisit  (airlines,  in  particular)  the  process  for  timeliness 
of  document  revision. 

Government  Actions 

•  Hold  manufacturers,  airlines,  maintenance  organiza¬ 
tions,  and  individuals  responsible  for  inferior  documents 
that  they  knowingly  create  or  use. 

Individual  Actions 

•  Report  bad  documentation  when  you  see  it.  Demand 
that  it  be  addressed.  Actively  change  the  culture! 
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2.2.10  Improve  coordination  of  document  profes¬ 
sionals  from  industry  segments  and  government 

The  workshop  participants  were  concerned  that  each 
manufacturer,  operator,  MRO,  and  FAA  office  seems  to 
be  individual  entities  with  limited  coordination.  There 
is  very  little  coordination  or  shared  information  among 
the  various  entities  that  create,  verify,  validate,  or  approve 
documentation.  The  group  suggests  that  something  like 
an  annual  conference  would  be  valuable  to  all  parties. 
This  is  integrally  tied  to  setting  appropriate  standards  for 
technical  documentation. 

Industry  Actions 

•  Promote  an  annual  gathering  of  documentation  pro¬ 
fessionals  from  within  company,  industry-wide,  and 
government.  Invite  expert  panelists  and  speakers/trainers 
to  help  standardize  values  and  approach. 

Government  Actions 

•  Commit  to  increasing  professional  training  and  confer¬ 
ence  attendance  for  FAA  inspectors  from  AEG  and  other 
documentation  roles. 

•  Create/revise  documents  and  guidance  to  promote 
education  and  standardization. 

Individual  Actions 

•  Individual  document  creators/engineers/regulators 
should  push  for  the  recommendations  made  in  this 
report. 

2.2.11  Challenges  and  Action  Item  Summary 

Technical  documentation  is  one  of  the  most  common 
problems  in  aviation,  and  the  solutions  are  extremely  dif¬ 
ficult  in  todays  typical  culture.  Addressing  the  problem 
will  require  significant  commitment  and  investment  by 
all  parties.  Fixing  it  means  changing  a  culture.  The  ques¬ 
tion  remains:  Is  the  aviation  industry  ready  to  tackle  these 
challenges? 

The  ten  prioritized  action  items  presented  here  to  ad¬ 
dress  technical  documentation  issues  are  not  dependent 
on  new  or  modified  regulations. 

The  manufacturers  are  willing  to  alter  procedures  to 
improve  effectiveness,  but  such  alteration  requires  an  ef¬ 
fective  reporting  system  that  informs  them  of  problematic 
procedures,  graphics,  etc. 

The  recommendations  can  proceed  as  quickly  as  industry 
and  government  are  willing  to  invest  resources  in  the  issue. 
The  recommendations  can  and  should  proceed  in  parallel. 

The  FAA  has  initiated  activity  by  submitting  techni¬ 
cal  documentation  as  an  important  research  activity  for 
2012-2015  funding.  FAA  divisions  representing  both  GA 
and  scheduled  air  carriers/MRO  submitted  requests  for 
support  in  this  area. 
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